

WHAT IS CLAIMED IS: 



1. A gam^ apparatus with which a player plays a game 
with virtual Object (s) in a mixed reality space 
comprising : \ 
5 viewpoint detection means for detecting the 

location/posture orya viewpoint of the players- 
geometric information, acquisition means for acquiring 
geometric information q± real oSject{s);^ "~ ~ 

recognition means ^or recognizing a current, 
relative relationship betwteen the virtual object (s) and 
real object (s); \ 

a rule memory for storing rule;^ for controlling the 
action of the virtual object {s)\ 

computation means for deterinining the next action 
of the virtual object (s) in accordarice with the rules 
stored in said rule memory and in correspondence with 
the location/posture of the real objecit(s), and 
computing thee location/posture of the Virtual 
object (s) after the determined action; and 

presentation means for generating at\ least one of 
image of the virtual object on the basis of\the 
location/posture of the virtual object (s) af^ier the 
action and the location/posture of the viewpoiVt 
position of the player, and for representing th^ mixed 
25 reality space to the player by superimposing the\ 
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virtual object image (s) on the player's view of the 
real space, 
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2. A game apparatus according to claim 1, wherein 
said presentation means further comprising, 

image-captuising means for capturing real space 
images of said playi^r ' s view of the real space; 

image generatidn means for generating xruTxed ' 
reality images represerating of the mixed reality space 
by superimposing or overvLaying said virtual object 
image (s) on said real spacse images; and 

a video see-through type display means that the 



player wares wherein said mixed reality images are 
displayed. 



3 . 



A game apparatus according to claim 1, wherein 
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said presentation means further comprising. 

An optical see- through type dn\splay means that 
the player wares wherein said virtual^^ object image (s) 
are displayed. 



4. A game apparatus according to clainj 1, further 
comprising, 

status detecting means for detecting ^tatus of 
2 5 the player; 
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whe^rein said computation means determines a next 
action of Vhe virtual object in accordance with the 
rule stored said rule memory and in correspondence 
with the loca\ion/posture of the real object and/or the 
status, and commuting a location/posture of the virtual 
object after the xietermined action. 

5. The apparatus^ according to claim 1, wherein the 
current, relative relationship includes a layout 
relationship between tl^e virtual object and real object 
at that time in the mixeid reality space. 

6. The apparatus according to claim 1, wherein the 
current, relative relationsnsip includes a behavior of 
the real object with respect the virtual object at 
that time in the mixed reality !5pace . 

7. The apparatus according to Vlaim 1, wherein the 
real object includes the player himself or herself, and 
said recognition means recognizes aVurrent, relative 
relationship between the virtual objeGit and the player. 

8. The apparatus according to claim V, wherein the 
real object includes a plurality of playWs who operate 
said game apparatus, and the plurality ofNplayers share 
a single mixed reality space. \ 

9. The apparatus according to claim 1, wherein the 
real object is an object which is fixed in posation, 
and \ 



saYd geometric information acquisition means 
comprises : 

a prec^termined memory for pre-storing location 
information a\^d shape information of the real object; 
and 

means for reading out the location information 
and shape information of the real object from said 
memory as needed, 

10 . The apparatus according to claim 1 , wherein the 
real object is an objec\ which is movable but does not 
deform, and 

said geometric infori\ation acquisition means 
comprises : 

a predetermined memory fvpr pre-storing shape 
information of the real obj ect ;^ 

a location/posture sensor for detecting a 
location/posture of the real obj eot ; and 

means for setting a region the real object is 
expected to occupy in the mixed realXspace in 
accordance with the detected location/vposture of the 
real object . 

11 . The apparatus according to claim l\ wherein the 
real object is a player, and 

said geometric information acquisiti(3^ means 
comprises : 



a sensor for detecting a location/posture of a 
head of thet player; and 

means Xp^ setting a region having a fixed, known 
shape that applroximates the player in the mixed reality 
5 space in accordai;ice with the detected location/posture 
of the head. 

12. The apparatus \according to claim 1, wherein when 
the game is a battle grame with the virtual object, an 
fg objective is to decrease an expected score of the 

10 player. 

m 

Z 13. The apparatus accordi^ng to claim 1, wherein when 

the game is a cooperative game with the virtual object. 



an objective is to increase an\ expected score of the 
player . 



12 15 14. The apparatus according to claim 1, wherein the 

rule controls the action of the virtual object on the 
basis of an objective of the game and a relative 
relationship between the virtual object and real object. 

15. The apparatus according to claim \l, wherein the 
20 rule stored in said rule memory expresses the action of 

the virtual object as an action pattern with a 
predetermined aim for achieving the objective. 

16. The apparatus according to claim 15, wherein the 
pattern has a path disadvantageous to the player in 

25 consideration of a layout relationship betweer\ the 
virtual object and real object, 
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17. The Jjpparatus according to claim 11, wherein when 
the player \s one of real objects, an output from said 
viewpoint defection means for detecting the 
location/posture of the viewpoint of the player is also 
used as informaVion which is to be acquired by said 
geometric information acquisition means and pertains to 
a location and shaj^e of the player. 

18. The apparatus ^according to claim 1, wherein said 
viewpoint detection m^ans detects a location/posture of 

10 a head of the player, ^d 

said apparatus furMier comprises detection means 
for detecting a location/p9sture of a hand of the 
player; and 

means for recognizing a\ relative location of the 
15 hand of the player with respecV to the head as a 

command on the basis of an outpi\t from said detection 
means . 

19. The apparatus according to c\aim 1, wherein said 
presentation means comprises: 

20 means for aligning the locationVposture of the 

real object to the location/posture of\ the virtual 
object after movement; 

means for generating an image of tfte virtual 
object after alignment in correspondence v^ith an 
25 occlusion relationship; and 

a head-mounted display device, 
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20. AnXimage processing method for a game apparatus 
with whicnv a player plays a game with virtual object (s) 
in a mixed Reality space comprising: 

viewpoint detection step for detecting the 
location/postuVe of a viewpoint of the player; 

geometric Ninf ormation acquisition step for 
acquiring geometric information of real object (s); 
- recognition step for recognizing a current, 

relative relationshi©^ between the virtual object (s) and 
real object (s); \ 

a rule memory forXstoring rules for controlling 
the action of the virtua^v object (s) ; 

computation step f orXdetermining the next action 
of the virtual object (s) in \accordance with the rules 
stored in said rule memory and in correspondence with 
the location/posture of the reaJ. object (s), and 
computing thee location/posture V)f the virtual 
object (s) after the determined action; and 

presentation step for generating at least one of 
image of the virtual object on the oasis of the 
location/posture of the virtual objecV(s) after the 
action and the location/posture of the\ viewpoint 
position of the player, and for representing the mixed 
reality space to the player by super imposdng the 
virtual object image (s) on the player's vi^ew of the 
real space . \ 



21. The method according to claim 20, wherein the 
player warea a video see- through type display, and said 
presentation Nstep further comprising, 

image-capturing step for capturing real space 
images of said ©layer's view of the real space; 

image genef^ation step for generating mixed 
reality images representing of the mixed reality space 
by superimposing or\ overlaying said virtual object 
image (s) on said reaJS. space images and for displaying 
said mixed reality imaiges on the display. 

22. The method according to claim 20, wherein the 
player wares an optical see-through type display and 
said presentation step repVesenting the mixed reality 
space to the player by displaying the virtual object 
image (s) on the display. \ 

23. The method according to Vlaim 20, further 
comprising, \ 

status detecting step for detecting status of the 
player ; \ 

wherein said computation stero determines a next 
action of the virtual object in accordance with the 
rule stored in said rule memory and Vn correspondence 
with the location/posture of the real\ object and/or the 
status, and computing a location/postuVe of the virtual 
object after the determined action. \ 
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24. toie method according to claim 20, wherein the 
current ,\relative relationship includes a layout 
relationship between the virtual object and real object 
at that time\^^in the mixed reality space. 

25. The method according to claim 20, wherein the 
current, relative relationship includes a behavior of 
the real object wi^h respect to the virtual object at 

"that t:ime' in "the ~m^i?ced real space . 

26. The method accolrding to claim 20, wherein the 
real object includes tt^ player himself or herself, and 
the recognition step incVides the step of recognizing a 
current, relative relationVhip between the virtual 
object and the player. \ 

27. The method according toNclaim 20, wherein the 
real object includes a plurality of players who operate 
the game apparatus, and the plurality of players share 
a single mixed reality space. \ 

28. The method according to claimyO, wherein the 
real object is an object which is fixted in position, 
and \ 

the geometric information acquisiMon step 
includes the steps of: \ 

pre-storing location information andXshape 
information of the real object in a predeteismined 
memory; and ^ 



reading out the location information and shape 
inf ormatiori of the real object from the memory as 
needed . \ 

29. The metnpd according to claim 20, wherein the 
real object is Vn object which is movable but does not 
deform, and \ 

the geometricv information acquisition step 
TincXudes ~the 'steps of: - 

pre-storing shape information of the real object 
in a predetermined memoVy; 

detecting a location/posture of the real object 
by a location/posture sens(s>r; and 

setting a region the real object is expected to 
occupy in the mixed real spac^ in accordance with the 
detected location/posture of the real object. 

30. The method according to cl)aim 20, wherein the 
real object is a player, and \ 

the geometric information acquisition step 
includes the steps of: \ 

detecting a location/posture of\a head of the 
player; and \ 

setting a region having a fixed, known shape that 
approximates the player in the mixed reality space in 
accordance with the detected location/posture of the 
head. \ 



31. 'Mne method according to claim 20, wherein when 
the gameVis a battle game with the virtual object, an 
objective Nis to decrease an expected score of the 
player. \ 

32. The method according to claim 20, wherein when 
the game is a cooperative game with the virtual object, 
an objective is ts.o increase an expected score of the 
"player. \ 

33. The method accSDrding to claim 20, wherein the 
rule controls the action of the virtual object on the 
basis of an objective o^ the game and a relative 
relationship between the Virtual object and real object. 

34. The method accordingxto claim 20, wherein the 
rule stored in the rule memolry expresses the action of 
the virtual object as an action pattern with a 
predetermined aim for achievingX the objective, 

35. The method according to clasim 34, wherein the 
pattern has a path disadvantageousX to the player in 
consideration of a layout relationship between the 
virtual object and real object, \ 

36. The method according to claim 3 0\ wherein when 
the player is one of real objects, an oWput from said 
viewpoint detection step of detecting the 
location/posture of the viewpoint of the player is also 
used as information which is to be acquire<^ in the 
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geometric information acquisition step and pertains to 
a location and shape of the player . 

37. The method according to claim 20, wherein the 
viewpoint deVection step includes the step of detecting 
a location/posture of a head of the player, and 

said method further comprises the detection step 
of detecting a looation/posture of a hand of the 
player; and \ 

the step of reccsgnizing a relative location of 
the hand of the player Vith respect to the head as a 
command on the basis of an output in the detection step. 

38. The method accordingXto claim 20, wherein the 
presentation step includes the steps of: 

aligning the location/posture of the real object 
to the location/posture of the Virtual object after 
movement ; and \ 

generating an image of the virtual object after 
alignment in correspondence with anXocclusion 
relationship . \ 

39. A storage medium which stores a program of an 
image processing method for a game apparatus with which 
a player plays a game with virtual objecV(s) in a mixed 
reality space comprising: \ 

viewpoint detection program step for Nietecting 
the location/posture of a viewpoint of the pVayer; 
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geometric information acquisition program step 
for acquiring geometric information of real object (s); 

recogniVion program step for recognizing a 
current, relatWe relationship between the virtual 
5 object (s) and reVl object(s); 

a rule memoiry for storing rules for controlling 
the action of the v^tual object(s); 

c6mputati'on"prog^^ step for determining the next 
action of the virtual object (s) in accordance with the 
10 rules stored in said rule, memory and in correspondence 
with the location/posture W the real object(s), and 
computing thee location/postyre of the virtual 
object(s) after the determined action; and 

presentation program stem for generating at least 
15 one of image of the virtual objeort on the basis of the 
location/posture of the virtual oodect(s) after the 
action and the location/posture of \he viewpoint 
position of the player, and for representing the mixed 
reality space to the player by super infoo sing the 
2 0 virtual object image (s) on the player ' s\ view of the 
real space. 
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